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(54) Motion compensated image interpolation 

(57) A motion compensated image interpolation system is described in which a mechanism is provided for 
the adjustment of control values to the apparatus to reduce artifacts in the interpolation output images. Output 
interpolated images that are substantially temporally coincident with input images are subtracted from the 
input images. If the interpolation process is accurate, then there should be no difference. A difference that is 
present can form the basis of an error signal that may be used in a feedback process to adjust control inputs to 
the apparatus. In some embodiments the interpolation images that are temporally coincident with input 
images will be detected as and when they occur for use in performing this analysis. In other embodiments, the 
output interpolated images can undergo an additional interpolation process to be projected back to the input 
image times to provide a continuous error signal. The error signal may control parameters such as a threshold 
used in alias rejection in correlation surface analysis or search block size control in correlation surface 
generation. 
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At least one drawing originally filed was informal and the print reproduced here is taken from a later filed formal copy. 
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MOTION COMPENSATED IMAGE INTERPOLATION 

This invention relates to the field of motion compensated image 
interpolation. More particularly, this invention relates to motion 
compensated image interpolation in which output images at an output 
rate are generated from input images at a input rate that is different 
from the output rate. 

Motion compensated image interpolation for purposes such as 
standards conversion is known, e.g. from film to television or from one 
television format to another. An example of such a system is described 
in British Published Patent Application GB-A-2 ,231 , 7*49 (Sony 
Corporation) . Known motion compensated standards converters suffer 
from motion artifacts. These artifacts can be controlled by the 
adjustment of system parameters such as threshold and conversion modes. 
An operator is required to study the output video for errors and then 
steer or alter the system parameters accordingly. This is expensive 
and is not satisfactory for on-line operation. 

This invention addresses the problem of more effectively 
controlling the system parameters to improve output image quality. 

Viewed from one aspect this invention provides apparatus for 
generating motion compensated output images at an output rate from 
input images at an input rate, said output rate being different from 
said input rate, said apparatus comprising: 

an output interpolator for generating interpolated images at said 
output rate from temporally adjacent input images in dependence upon 
image motion vectors; 

means for subtracting at least a portion of a non motion 
compensated image, taken from a corresponding input image, from at 
least a portion of an substantially temporally coincident test 
interpolated image to form an error signal; and 

means for adjusting control inputs to said apparatus in 
dependence upon said error signal . 

The invention recognises that an automatic feedback path may be 
provided by comparing an interpolated image with a temporally 
coincident non motion compensated image taken from an input image. Any 
differences between these two images are likely to be due to any 
artifacts introduced by the motion compensated interpolation process. 



Thus, deriving an error signal from the difference between the two 
images provides a measure of the amount of artifacts in the image that 
can then be used as a feedback signal to adjust control inputs to the 
apparatus. The input images effectively used as reference signals 
which the motion compensated interpolation process should be able to 
faithfully reproduce if it is free from artifacts. 

The additional hardware requirements needed to implement the 
invention can be advantageously reduced in preferred embodiments in 
which said corresponding input image is substantially temporally 
coincident with an interpolated image from said output interpolator, 
said test interpolated image being formed by said output interpolator. 

In a synchronous, or even an asynchronous, system with differing 
output and input rates, there will be times at which an output frame is 
substantially temporally coincident with an input frame. When this 
occurs, very little, if any, processing is required to produce the 
output frame from the input frame and the spare hardware capacity 
within the system can be utilised to perform the above described 
control operation. 

In a synchronous system, there are likely to be times of exact 
temporal coincidence, whereas in an asynchronous system some sort of 
thresholding is needed to establish that the frames are substantially 
temporally coincident to a degree at which a meaningful comparison can 
be made. A temporal coincidence of perhaps less than k% of the inter 
image period might be deemed acceptable. 

It will be appreciated that apparatus for performing motion 
compensated image interpolation is complex and frequently alters an 
input image in several different ways other than the basic application 
of motion vector compensated interpolation, e.g. filtering, 
subsampling, line rate conversion, interlaced to progressive scan 
conversion etc. In such situations, the interpolator has an effect 
upon the input image even if no motion vectors are applied. In order 
to take account of this so as to not produce inappropriate error 
signals due to other than motion artifacts, preferred embodiments are 
such that when said output interpolator has a zero motion filtering 
action, said non motion compensated image is an input image subject to 
said zero motion filtering action. 

As previously mentioned, it is important that there should be 



some mechanism for determining when the non motion compensated image 
and corresponding input image are substantially temporally coincident. 
A preferred way of dealing with this is to provide a threshold detector 
for detecting a corresponding input image that is substantially 
temporally coincident with an interpolated image from said output 
interpolator as one in which a motion vector of maximum magnitude for 
said interpolated image multiplied by a value of temporal displacement 
of said interpolated image from said input image is below a threshold 
value . 

Providing a threshold detector operating in a manner responsive 
to the maximum magnitude of motion vector within a particular scene 
allows the system to take account of the degree of motion present when 
assessing whether the test interpolated image and the corresponding 
input image are sufficiently temporally close to be deemed temporally 
coincident, i.e. if a large amount of motion is present, then the 
images have to be temporally closer so that the comparison is not 
distorted by genuine differences between the images due to the motion. 

The data processing requirements for a system that performs 
motion compensated image interpolation are high. The amount of image 
data to be handled and the number of calculations to be performed is 
typically large. In order to deal with this, it is important that the 
apparatus should operate at high speed and be efficient. In preferred 
embodiments of the invention said interpolator includes: 

a first multiplier for multiplying pixel values from a first 
input image by a first temporal weighting coefficient to produce a 
first partial interpolated pixel value; 

a second multiplier for multiplying pixel values from a second 
input image by a second temporal weighting coefficient to produce a 
second partial interpolated pixel value; and 

an adder for adding said first partial interpolated pixel value 
and said second partial interpolated pixel value to produce an 
interpolated pixel value. 

With the above form of interpolator for providing the normal 
interpolation action, it is possible to also use this interpolator to 
act as the means of subtracting. Accordingly, in preferred embodiments 
said means for subtracting comprises said first multiplier for 
multiplying pixel values from said non motion compensated image by a 



first subtraction coefficient, said second multiplier for multiplying 
pixel values from said corresponding image by a second subtraction 
coefficient that is of equal magnitude and opposite sign to said first 
subtraction coefficient and said adder for adding outputs from said 
first multiplier and said second multiplier to form said error signal. 

It will be appreciated that the requirement for the subtraction 
operation is that the first and second subtraction coefficients should 
be of equal magnitude and opposite sign. An advantageous reduction in 
the amount of processing required is achieved if these coefficients 
have a magnitude of unity. 

A convenient system for identifying the motion vectors to be used 
in the motion compensated image interpolation process is to provide: 

means for calculating a correlation surface representing image 
correlation between a search block of image data within a first input 
image and portions of a temporally adjacent second input image 
displaced by differing displacement vectors from said search block; and 

means for detecting a motion vector corresponding to a point of 
correlation maximum in said correlation surface. 

In such systems, it is usual to provide a mechanism for 
identifying a correlation maximum in a correlation surface that 
provides a degree of resistance to problems such as aliasing in the 
correlation surface. Various tests can be used to this end, their 
common feature being the provision of some sort of threshold 
correlation maximum confidence value. The adjustment of this threshold 
value is a system control parameter that can significantly alter the 
number of artifacts introduced by the system. Accordingly, in 
preferred embodiments of the invention said error signal is low pass 
spatially filtered, said means for adjusting serving to control a 
threshold correlation maximum confidence value, used by said means for 
detecting to reject a correlation maximum that is insufficiently 
clearly characterised, in dependence upon said low pass spatially 
filtered error signal. 

Another system control parameter that has an effect upon the 
quality of the output is the size of the vector selection block match 
block used. Accordingly, in preferred embodiments said error signal is 
high pass spatially filtered, said means for adjusting serving to 
control search block size in dependence upon said high pass spatially 



filtered error signal. 

It will be appreciated that motion compensated image 
interpolation systems are complex and the manner in which a control 
parameter should be adjusted to reduce the artifacts is not immediately 
apparent. If the detected error signal increases, it may be that for 
some images an increase in vector selection block match block size is 
required whereas for other images a decrease is required. In order to 
deal with this, in preferred embodiments said means for adjusting 
performs test adjustments to a control input to produce a first test 
error signal from a test higher control input value and a second test 
error signal from a test lower control input value, said first test 
error signal, said second test error signal and a current error signal 
from a current control input value being compared to identify that 
control input value to be adopted as yielding a lower error. 

In this way, the means for adjusting adopts a trial-and-error 
approach to determine which way a control input value should be 
adjusted. 

One way of operating such a trial-and-error system is to rely 
upon the relative infrequency of scene changes and slow change within 
a scene and provide that said means for subtracting forms said first 
test error signal, said second test error signal and said current error 
signal from three sequential non motion compensated image and test 
interpolated image pairs. These test error signals are compared and a 
decision made as to the best system parameter before a change to the 
system parameter used for generating output images is made. 

Thus, if temporal coincidence occurs at every five output frames, 
then the original error signal will be derived at output frame 0, the 
first test error signal at output frame 5 and the second test error 
signal at output frame 10. 

An alternative approach is to derive the current error signal, 
the first test error signal and the second test error signal at the 
same time. This may be achieved in embodiments in which said means for 
subtracting forms said first test error signal, said second test error 
signal and said current error signal from a repeated portion of a 
single non motion compensated and test interpolated image pair. 

Each of the error signals is derived from the same sub-portion of 
the image so that differences do not arise due to image scene 



differences. Thus, in some embodiments a central third of an image 
scene may be used for each of the three error signal derivations . 

On a more general level, it will be appreciated that the error 
signals and the subtraction analysis described above may be performed 
over a portion of the images concerned rather than the full images even 
outside of the context of this particular preferred feature. 
Furthermore, the invention is equally applicable to both frame based 
and field based systems. 

As previously discussed, one class of embodiments of the 
invention perform the subtraction to derive the error signal at times 
when an output interpolation image happens to coincide with an input 
image. In another class of embodiments it may be preferred that there 
is provided a test interpolator for interpolating said test 
interpolated image from at least one interpolated image passed from 
said output interpolator. 

In this way, rather than waiting for the temporal coincidence to 
occur, the test interpolator generates a test interpolated image that 
is temporally coincident with an input image from those interpolated 
images produced by the output interpolator. Accordingly, the feedback 
control may be made continuously operable and faster acting. 

In some embodiments it may be advantageous to reduce the amount 
of hardware required by providing that said test interpolator acts in 
dependence upon those motion vectors used by said output interpolator. 
An alternative that is able to deal with errors that might otherwise be 
masked by the projection/interpolation back to the input image times is 
to provide that said test interpolator operates using a different 
algorithm from said output interpolator and detects its own motion 
vectors . 

In addition to the adjustment of control parameters for the 
threshold and search block size discussed above, a control parameter 
may be provided that serves to control reworking of an interpolated 
image where errors are detected. Accordingly, in preferred embodiments 
means for adjusting provides a control input that acts to switch 
generation of an error portion of an interpolated image for which said 
error signal is unacceptable to a further output interpolator that 
generates an alternative interpolated image portion to replace said 
error portion. 



With this preferred feature, should errors occur in a particular 
error portion of an image, then it is possible to use the feedback to 
switch those error portions to be reworked, possibly using different 
vectors or with different search block size so as to improve the image 
quality at the error portion. 

As discussed above, it is not always immediately apparent which 
way the control inputs to the system should be adjusted to improve 
image quality. This also applies to the reworking of the image. 
Accordingly, in preferred embodiments there is provided a further test 
interpolator for interpolating at least a further test interpolated 
image portion at a temporal position substantially coincident with said 
non motion compensated image using said alternative interpolated image 
portion; 

further means for subtracting at least a portion of said non 
motion compensated image from at least said further test interpolated 
image portion to form a further error signal; and 

means for controlling said switching in dependence upon a 
comparison of said error signal and said further error signal. 

The provision of the further test interpolator and further means 
for subtracting allows the quality of the reworked error portion to be 
compared with that originally produced and only substituted in its 
place should there be an improvement. 

When reworking error portions of an image, it is possible to 
introduce disturbing edge effects at the interface between the reworked 
and the non reworked portions that negate any improvement in image 
quality that is being sought. In order to reduce this, it is preferred 
that said alternative interpolated image portion from said further 
output interpolator and said interpolated image from said output 
interpolator are combined together using a weighting function centred 
upon said error block that decreases that contribution from said 
alternative interpolated image portion and increases that contribution 
from said interpolated image upon moving away from said error block. 

Viewed from another aspect this invention provides a method of 
generating motion compensated output images at an output rate from 
input images at an input rate, said output rate being different from 
said input rate, said method comprising the steps of; 

generating interpolated images at said output rate from 
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temporally adjacent input images in dependence upon image motion 
vectors; 

subtracting at least a portion of a non motion compensated image, 
taken from a corresponding input image, from at least a portion of an 
substantially temporally coincident test interpolated image to form an 
error signal; and 

adjusting control inputs to said apparatus in dependence upon 
said error signal. 

Embodiments of the invention will now be described, by way of 
example only, with reference to the accompanying drawings in which: 

Figure 1 illustrates the relative timing of input and output 
images ; 

Figure 2 illustrates apparatus for interpolating an output value 
between temporally adjacent input images; 

Figure 3 illustrates the use of the apparatus of Figure 2 to 
provide an error signal measuring system performance; 

Figure 4 illustrates a system of which the apparatus of Figures 
2 and 3 forms part; 

Figure 5 illustrates the image sequential trial-and-error 
approach to adjusting a control value; 

Figure 6 illustrates the concurrent trial-and-error approach to 
adjusting a control value; 

Figure 7 illustrates a problem that can occur in finding the best 
setting for a control value; 

Figure 8 illustrates a real time standards converter employing an 
alternative approach to that previously illustrated; 

Figure 9 illustrates a non-real time version of the embodiment of 
Figure 8; 

Figure 10 illustrates forward projection from output interpolated 
images to test interpolated images; 

Figure 11 . illustrates backward projection from output 
interpolated images to test interpolated images; 

Figure 12 illustrates a circuit for extracting an error signal 
utilising both forward and backward projection; 

Figure 13 illustrates a weighting function for the mixing 
together of reworked and non-reworked image portions; and 

Figure 14 illustrates a circuit utilising the weighting function 



of Figure 13 to mix together reworked and non-reworked image portions. 

Figure 1 illustrates the relative temporal positions of a 
sequence of input images 2 and output images 4. It will be seen that 
the group of six input images 6 maps to the group of five output images 
8. Such a mapping could correspond to a 60Hz to 50Hz format 
conversion . 

The case illustrated is a synchronous arrangement. In this case, 
each fifth output image 10, 12 is temporally coincident with a 
corresponding input image 14, 16. The relative temporal positions of 
the output images 4 within the period between two input images 2 is 
illustrated at the bottom of the figure. In this synchronous system, 
the positions are t=0. 0.2, 0.4 . 0.6 and 0.8. In a non-synchronous 
system the relative temporal positions would be unlikely to have such 
round number values, but nevertheless the same effect would be seen 
whereby in moving through the sequence of output images k , periodically 
an output image would arise that was substantially temporally 
coincident with a corresponding input image. 

For the output images that are not temporally coincident with 
input images, a motion compensated interpolation between the temporally 
adjacent input images is made to derive the input image. Considering 
the output image 18, this is formed by a forward projection of the 
vectors (indicated by +v) from input image 20 and a backward projection 
of the vectors (indicated by -v) from input image 22. The relative 
temporal position between the input images 20, 22 is t=0.2 and so the 
contribution from the input image 20 is weighted to have more account 
than that from the input image 22. This is a linear weighting whereby 
the contribution from input image 20 is multiplied by 0.8 and that from 
the input image 22 is multiplied by 0.2; other weightings, such as a 
cosine weighting, are possible. 

Figure 2 illustrates part of the apparatus for performing the 
motion compensated interpolation operation illustrated in Figure 1 . 
Temporally adjacent input images are input along the signal lines 
present video and next video. These are fed to respective vector 
application and interpolation units 2k, 26. The motion vectors 
identified between the temporally adjacent input images are also 
supplied to these units. When a particular pixel value in an output 
interpolated image is to be derived, the motion vector for that point 
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is applied as a backward projection in the vector application and 
interpolation unit 26 and a forward projection in the vector 
application and interpolation unit 24. These vector applications are 
also weighted depending upon the relative temporal position of the 
output interpolated image between the two input images as indicated by 
the vectors v.t and v(l-t). 

The pixel values pointed to by the backward and forward projected 
vectors within the input images are then fed to respective first and 
second multipliers 28, 30. These multipliers then weight the pixel 
values according to the temporal proximity of the output interpolated 
image (as indicated by the weighting coefficients (1-t) and t, before 
feeding their outputs to an adder 32 from which an output interpolated 
pixel value is supplied. 

Figure 3 illustrates a modification to the embodiment of Figure 
2 to derive an error signal. In this form of operation, the output 
interpolated image is substantially temporally coincident with the 
input image on the present video input line. Accordingly, the system 
output can be tapped from the output of the multiplier 28. This output 
has then been subject to the same filtering and other operations except 
the addition by the adder 32 that normally occur during motion 
compensated interpolation. Alternative taps for the system output 
could be taken from between the vector application and interpolation 
unit 24 and the multiplier 28 or, with the use of an equalising delay 
unit 3^t from upstream of the vector application and interpolation unit 
24. 

The motion vectors between the temporally adjacent input images 
on the present video signal line and the next video signal line are 
supplied to the vector application and interpolation units 24, 26. 
Since the temporal position t is substantially zero, the vector 
application and interpolation unit 24 has effectively zero motion 
compensation action. However, the vector application and interpolation 
unit 26 backward projects to effectively the full extent of the motion 
vectors to generate what should correspond to the system output (non 
motion compensated image). 

The multipliers 28 and 30 are supplied with respective weighting 
coefficients of +1 and -1. These coefficients, in conjunction with the 
action of the adder 32 serve to provide a means for subtracting a test 



interpolated image output from the multiplier 30 from a non-motion 
compensated image output from the multiplier 28. The output from the 
adder 32 corresponds to an error signal indicating the difference 
between what should have been interpolated and what was interpolated. 
If the interpolation was free from artifacts and entirely accurate then 
the error signal would be zero for all points within the output image. 
In reality, the error signal is non-zero. 

In- a synchronous system, which is forced to lock such that 
temporal positions of zero occur, this analysis can be performed on a 
regular basis, e.g. a 60Hz to 50Hz converter will have the zero 
temporal position every five output fields (100ms). In an asynchronous 
system, this technique can still be applied, but the zero position will 
occur infrequently. It is possible to increase the frequency of 
analysis at the expense of output temporal smoothness by treating 
temporal positions near t=0 (e.g. at approximately t<0.04) as if they 
were at t=0. A refinement of this approach is to access the highest 
magnitude motion vector for an output image (available from the earlier 
processing that extracts the motion vectors) and multiply this by the 
temporal position of the output image. The result is compared with a 
threshold value to determine whether the combined effects of the 
relative temporal position and the amount of motion present in the 
image indicate that the output is substantially temporally coincident 
with the input so as to yield a meaningful error signal. 

Figure 4 illustrates a system of which the apparatus of Figures 
2 and 3 forms part. The system comprises a standards converter 34 
including an interpolator stage 36 that provides a system output upon 
the line 38 formed of output interpolated images and images tapped from 
downstream of the multiplier 28 during analysis of system performance, 
as shown in Figure 3- The error signal of Figure 3 is fed to error 
filters 40 where it may be subject to various filtering operations, 
e.g. low pass spatial filtering and high pass spatial filtering. The 
filtering operations may be non-linear, for example, to perform 
thresholding to remove errors below a certain value, or exponential 
weighting to increase the contribution of high contrast errors. The 
output or outputs from the error filter 40 are fed to a summation unit 
42 where they are summed over a complete output image. A system 
performance figure (SPF) for that image can then be fed to a control 
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system that provides control inputs to the standards converter 34. 

The error field derived from the error signal can be analysed by 
low pass filtering to yield information on large areas of error. This 
can be used to control the threshold used in vector estimation so as to 
select how clearly defined a correlation maximum must be before it is 
used as a source of a motion vector. In addition, or alternatively, 
the error signal may be high pass filtered to yield information on 
small areas of error. This information can be used for the control of 
vector selection block match size used in motion vector identification 
of the known type. 

The direction in which a control parameter should be moved to 
reduce the amount of error present is not always apparent and amongst 
other things may depend upon the nature of the image currently being 
processed. Figures 5 and 6 illustrate two different approaches for 
dealing with this. Figure 5 relates to the conversion illustrated in 
Figure 1 whereby a point of temporal coincidence between an input image 
and an output image occurs for every fifth output image. Given that a 
control value is set to have a value a, this is used for output frame 
0 to derive a system performance SPF Qt at output frame 5. an alternative 
test control value a+<5 is tried and yields a system performance figure 
SPF-. This process is repeated at output frame 10 except that the 
control value this time is set to a-6. 

When these three system performance figures have been derived for 
sequential non-motion compensated and test interpolated image pairs, 
then the control value setting that gave the lowest error signal 
(system performance figure) can be chosen for subsequent use as 
indicated by the control value f>. 

Figure 6 illustrates an alternative approach whereby concurrent 
trial-and-error assessments are made. In this case, a central portion 
of the non-motion compensated and test interpolated image pair are 
compared. Since only a portion of the images are compared, processing 
capacity is freed up such that the three comparisons for a-6, a and a+<5 
can be made concurrently. This may be achieved by reading the 
appropriate pixel values from an input frame store through the 
apparatus of Figure 3 using read addresses that trace through the 
portion of the image to be tested. When the three trial-and-error 
comparisons have been made the new setting p" is chosen as that giving 
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the lowest system performance figure. 

Figure 7 illustrates a problem which may occur in a system in 
which more than one minimum is present in the function of system 
performance figure versus system setting being controlled. In such a 
case, it is possible that the system will identify and hold itself at 
a system performance figure minimum that is not the lowest that could 
be achieved. In order to counteract this, the system may be arranged 
to periodically choose a trial-and-error setting that differs 
considerably from that currently being used in an attempt to break away 
from any false minima that may currently be being used. 

Figure 8 illustrates an alternative embodiment in which a test 
interpolator is provided for interpolating the test interpolated image 
from at least one interpolated image passed from the output 
interpolator. In this way analysis of the system performance can be 
continuously monitored rather than only when an output image coincides 
with an input image. Thus, an input video signal is supplied to a main 
standards converter 46 that includes the output interpolator that acts 
as the primary interpolator for the system. The input video signal is 
also passed via equalizing delay units 48, 50 to an error analysis unit 
52 and an error reduction unit 54. The error analysis unit 52 is 
passed the motion vectors for each pixel used by the main standards 
converter 46 in synchronism with the arrival of the input video via the 
equalising delay unit 48. The error analysis unit 52 also receives the 
interpolated images output from the mains standards converter 46. 

These interpolated images are projected to be temporally 
coincident with the input images from which they were derived using the 
supplied motion vectors. If the interpolation processes have been free 
of artifacts, then the original input images should be regenerated from 
the interpolated images output by the main standards converter 46 and 
no difference will be observed when the error analysis unit 52 
subtracts these images. When artifacts are present, then the error 
signal produced by the subtraction in the error analysis unit 52 
indicates this. 

The error reduction unit 54 receives the input image via the 
equalizing delay units 48, 50 and serves to perform a further output 
interpolation in an attempt to produce a better result for the 
particular parts of the image concerned than was achieved by the main 
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standards converter 46. The error analysis unit 52 identifies the 
appropriate error portions of the image in which errors have occurred. 
This information is then used elsewhere in the circuit to generate 
alternative parameter control values that have been appropriately 
adjusted on the basis of the trial-and-error analysis. 

The error portion which the error reduction unit 54 has attempted 
to repair in addition to a corresponding selected portion of the input 
video is supplied to a secondary error analysis unit 56. This 
secondary error analysis unit 56 includes a further test interpolator 
for projecting the attempted repair patch for the error portion back 
onto the corresponding portion of the input video from which it is then 
subtracted by a further means for subtracting analogous to that in the 
main error analysis unit 52. The difference between these two signals 
forms a further error signal that may be compared with the original 
error signal from the error analysis unit 52 to determine whether the 
attempted repair by the error reduction unit 54 has produced a better 
or worse result. Control values for the main standards converter 46 
may also be adjusted by an output from the secondary error analysis 
unit 56. 

A mixer 58 receives both the original output interpolated images 
from the main standards converter 46 via a temporal delay line 60 and 
the repair patches for the error portions from the secondary error 
analysis unit 56. If the repair portions are of better quality than 
those produced by the main standards converter 46, then the mixer 58 
selects these for output. Otherwise, the output is taken from the main 
standards converter 46. 

The overall operation of the embodiment of Figure 8 is such that 
the main standards converter 46 produces motion compensated output 
video. This output video is fed to the error analysis unit 52 along 
with the vector selected for each output pixel. The error analysis 
unit 52 then generates video {by extrapolating the video and vectors) 
coincident with the "input" video. This video may be referred to as a 
reference video. The reference video is then compared to the actual 
input video, the areas of significant difference indicating where the 
main standards converter 46 is generating errors. 

The areas flagged in the error analysis unit 52 can then be 
reworked (with changed or steered parameters from those originally 



15 

used) in the error reduction unit 54 that is effectively a second 
smaller standards converter. An example of a parameter that could be 
changed is the threshold used in vector estimation. An example of a 
steered parameter could be that those vectors at pixel positions that 
caused errors are masked off from the vector selection process when 
this is repeated within the error reduction unit 54. 

The reworked video from the error reduction unit 5*1 is fed to a 
secondary error analysis unit 56 where it is used to generate reference 
video that can be compared to the input video. If this reworked video 
produces less errors than the original output, then it is passed to the 
output mixer 58. The parameter changes that have produced this 
improved output may also be passed to the main standards converter 46. 

The output mixer 58 combines the reworked areas of the video with 
the original output from the main standards converter 46. 

The size of the error reduction unit 54 need not be prohibitive. 
If typically only 10% of the picture at worst needs reworking, then the 
error reduction unit 54 need only be designed to cope for 10% of the 
data that the main standards converter 46 handles. The error reduction 
unit 54 can preferably be made to operate with completely different 
algorithms from that of the main standards converter 46, for example 
the vector estimation process in the error reduction unit 54 can be 
phase correlation based while that in the main standards converter 46 
can be block matched based. 

Figure 9 illustrates an off-line version of the embodiment of 
Figure 8. Video tape recorders 62. 64 provide storage for input 
images and output images from the system. The input images to be 
interpolated are fed to a main standards converter 66 via an input 
temporal delay unit 68 that is able to store at least two input images. 
The main standards converter 66 interpolates the input images to output 
images that are fed to an output mixer 70. The input images and the 
interpolated output images are also fed to an error analysis unit 72 in 
addition to the vectors for each output pixel. 

The error analysis unit 72 projects the output interpolated 
images back to the input image temporal positions and performs a 
subtraction to derive an error signal indicative of artifacts in the 
output interpolated images. If such artifacts are found, then the 
error analysis unit 72 controls the main standards converter 66 to 
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rework those error portions using the input images temporarily stored 
in the input temporal delay unit 68. These reworked portions can then 
be mixed with the rest of the image in the output mixer unit 70 before 
being fed to the output video tape recorder 6U. 

The embodiment of Figure 9 can be considered as a simplified 
version of that required for on-line generation illustrated in Figure 
8. The input to the system is processed by the mains standards 
converter 66, whose output is fed to the error analysis unit 72 and the 
output mixer 70. 

The error analysis unit 72 uses the interpolated image video and 
the vectors it receives for each output image pixel to generate 
reference frames of video that are compared to the input image video. 
If after this comparison an error signal exceeding a specified 
threshold is found, then the corresponding error portions of the output 
video are reworked. 

When areas are to be reworked, the input image video required is 
replayed through the main standards converter 66 either from the video 
tape recorder 62 or from the optional input temporal delay unit 68. 
The main standards converter 66 now performs the function of the error 
reduction unit 5^ of Figure 8. In this case, the mains standards 
converter 66 is designed to cope with full sized video data and so when 
working as an error reduction unit many more parameter variations can 
be tried within the capacity of the hardware. If it is assumed that 
only 10% of the video at worst may require reworking, then the capacity 
of the main standards converter 66 would mean that ten variants of the 
system parameters could be tried to find one that produced better 
results. When the video has been reworked by the main standards 
converter 66, it re-enters the error analysis unit 72 where it is 
interpolated back to the input image temporal positions and used to 
derive an error signal. If the reworked video is better than the 
original video, then it is selected and mixed with that previously 
stored in the output mixer 70. The parameters changes that have 
produced this improved output can then be adopted by the main standards 
converter 66 for that portion of the image. In this way it is possible 
to operate the convertor with different system control parameters for 
different portions of the image to be converted on the basis of what 
provides the best results. 
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Figures 10 and 11 illustrate the generation of reference 
images/test interpolated images from the output interpolated images. 
In Figure 10, the output interpolated images are forward projected to 
times coincident with input images. In Figure 11, the output images 
are backward projected to times coincident with input images. 

Figure 12 illustrates a circuit for deriving the error signal 
using forward and backward projected images. The input image video is 
fed to the standards converter ~jk where the output interpolated images 
are generated. The forward and backward projection units 76, 78 then 
perform the respective processes illustrated in Figures 10 and 11 to 
produce the test interpolated images. Sub tractors 80, 82 produce a 
difference between these reference images projected to temporally 
coincident positions with the input images using the input image video 
signal fed via an equalising delay unit 84. The outputs from the 
subtractors 80, 82 are fed to filter and summation units 86, 88 where 
they are subject to the sort of filtering operations previously 
discussed in relation to the embodiments of Figures 2 to k prior to 
being combined by an adder 90 to yield the error signal. 

Figure 13 illustrates a weighting function that can be used to 
mix together the originally interpolated and the reworked interpolated 
image in the mixers 58, 70 of Figures 8 and 9 to avoid noticeable edge 
effects. The weighting functions varies between a value of unity at 
the centre of the error portion to a value of zero far from the error 
portion. The weighting function is dependent upon the x, y coordinates 
measuring the displacement from the centre of the error portion. 

Figure 14 illustrates the operation of the mixers 58, 70 of 
Figures 8 and 9. The original and reworked blocks are fed to 
respective multipliers 92, 94 where they are multiplied by the 
complement of the weighting function and the weighting function 
respectively prior to being passed to an adder 96. At the centre of 
the reworked block, the output is derived fully from the reworked 
block. Distant from the centre of the reworked block, the output is 
derived fully from the original block. In this way, a smooth 
transition between the reworked data and the original data can be 
provided . 
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CLAIMS 

1 . Apparatus for generating motion compensated output images at an 
output rate from input images at an input rate, said output rate being 
different from said input rate, said apparatus comprising: 

an output interpolator for generating interpolated images at said 
output rate from temporally adjacent input images in dependence upon 
image motion vectors; 

means for subtracting at least a portion of a non motion 
compensated image, taken from a corresponding input image, from at 
least a portion of an substantially temporally coincident test 
interpolated image to form an error signal; and 

means for adjusting control inputs to said apparatus in 
dependence upon said error signal. 

2. Apparatus as claimed in claim 1, wherein said corresponding input 
image is substantially temporally coincident with an interpolated image 
from said output interpolator, said test interpolated image being 
formed by said output interpolator. 

3. Apparatus as claimed in any one of claims 1 and 2, wherein said 
output interpolator has a zero motion filtering action, said non motion 
compensated image being an input image subject to said zero motion 
filtering action. 

4. Apparatus as claimed in any one of claims 2 and 3. comprising a 
threshold detector for detecting a corresponding input image that is 
substantially temporally coincident with an interpolated image from 
said output interpolator as one in which a motion vector of maximum 
magnitude for said interpolated image multiplied by a value of temporal 
displacement of said interpolated image from said input image is below 
a threshold value. 

5. Apparatus as claimed in any one of claims 2, 3 and k, wherein 
said interpolator includes: 

a first multiplier for multiplying pixel values from a first 
input image by a first temporal weighting coefficient to produce a 
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first partial interpolated pixel value; 

a second multiplier for multiplying pixel values from a second 
input image by a second temporal weighting coefficient to produce a 
second partial interpolated pixel value; and 

an adder for adding said first partial interpolated pixel value 
and said second partial interpolated pixel value to produce an 
interpolated pixel value. 

6. Apparatus as claimed in claim 5. wherein said means for 
subtracting comprises said first multiplier for multiplying pixel 
values from said non motion compensated image by a first subtraction 
coefficient, said second multiplier for multiplying pixel values from 
said corresponding image by a second subtraction coefficient that is of 
equal magnitude and opposite sign to said first subtraction coefficient 
and said adder for adding outputs from said first multiplier and said 
second multiplier to form said error signal. 

7. Apparatus as claimed in any one of the preceding claims, 
comprising: 

means for calculating a correlation surface representing image 
correlation between a search block of image data within a first input 
image and portions of a temporally adjacent second input image 
displaced by differing displacement vectors from said search block; and 

means for detecting a motion vector corresponding to a point of 
correlation maximum in said correlation surface. 

8. Apparatus as claimed in claim 7, wherein said error signal is low 
pass spatially filtered, said means for adjusting serving to control a 
threshold correlation maximum confidence value, used by said means for 
detecting to reject a correlation maximum that is insufficiently 
clearly characterised, in dependence upon said low pass spatially 
filtered error signal. 

9. Apparatus as claimed in any one of claims 7 and 8, wherein said 
error signal is high pass spatially filtered, said means for adjusting 
serving to control vector selection block match block size in 
dependence upon said high pass spatially filtered error signal. 
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10- Apparatus as claimed in any one of the preceding claims, wherein 
said means for adjusting performs test adjustments to a control input 
to produce a first test error signal from a test higher control input 
value and a second test error signal from a test lower control input 
value, said first test error signal, said second test error signal and 
a current error signal from a current control input value being 
compared to identify that control input value to be adopted as yielding 
a lower error. 

11. Apparatus as claimed in claim 10, wherein said means for 
subtracting forms said first test error signal, said second test error 
signal and said current error signal from three sequential non motion 
compensated image and test interpolated image pairs. 

12. Apparatus as claimed in claim 10, wherein said means for 
subtracting forms said first test error signal, said second test error 
signal and said current error signal from a repeated portion of a 
single non motion compensated and test interpolated image pair. 

13. Apparatus as claimed in claim 1, comprising a test interpolator 
for interpolating said test interpolated image from at least one 
interpolated image passed from said output interpolator. 

14. Apparatus as claimed in claim 13, wherein said test interpolator 
acts in dependence upon those motion vectors used by said output 
interpolator . 

15. Apparatus as claimed in claim 13. wherein said test interpolator 
operates using a different algorithm from said output interpolator and 
detects its own motion vectors. 

16. Apparatus as claimed in any one of the preceding claims, wherein 
said means for adjusting provides a control input that acts to switch 
generation of an error portion of an interpolated image for which said 
error signal is unacceptable to a further output interpolator that 
generates an alternative interpolated image portion to replace said 
error portion. 
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17. Apparatus as claimed in claim 16, comprising: 

a further test interpolator for interpolating at least a further 
test interpolated image portion at a temporal position substantially 
coincident with said non motion compensated image using said 
alternative interpolated image portion; 

further means for subtracting at least a portion of said non 
motion compensated image from at least said further test interpolated 
image portion to form a further error signal; and 

means for controlling said switching in dependence upon a 
comparison of said error signal and said further error signal. 

18. Apparatus as claimed in any one of claims 16 and 17. wherein said 
alternative interpolated image portion from said further output 
interpolator and said interpolated image from said output interpolator 
are combined together using a weighting function centred upon said 
error block that decreases that contribution from said alternative 
interpolated image portion and increases that contribution from said 
interpolated image upon moving away from said error block. 

19- A method of generating motion compensated output images at an 
output rate from input images at an input rate, said output rate being 
different from said input rate, said method comprising the steps of: 

generating interpolated images at said output rate from 
temporally adjacent input images in dependence upon image motion 
vectors ; 

subtracting at least a portion of a non motion compensated image, 
taken from a corresponding input image, from at least a portion of an 
substantially temporally coincident test interpolated image to form an 
error signal; and 

adjusting control inputs to said apparatus in dependence upon 
said error signal . 

20. Apparatus for generating motion compensated output images 
substantially as hereinbefore described with reference to the 
accompanying drawings. 



21. A method of generating motion compensated output images 
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substantially as hereinbefore described with reference to the 
accompanying drawings. 
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